Let’s introduce/review raking!

We have a non-representative sample from a larger universe, which we want to use to make inferences about that universe. How do we best assign weights to each observation in the sample to make (weighted) estimates representative?

In Easy Mode:

• Our outcome \(Y^N\) is correlated with demographics (even in easy mode)
• We’ll have Ignorable Non-Response (conditional on weighting vars)
• More specifically, our sample happens to only need adjustment on age and race to match gen pop on this outcome
• We somehow know everything we need to weight on to estimate \(Y^N\)
• We also somehow know age and race proportions of the sample and universe completely without error

Check distribution

universe %>% summarize(mean_p = mean(enjoys_pineapple))

# A tibble: 1 × 1
  mean_p
   <dbl>
1  0.467

universe %>%
  group_by(age,race) %>%
  summarize(mean_p = mean(enjoys_pineapple)) %>%
  arrange(mean_p)

# A tibble: 10 × 3
# Groups:   age [2]
   age   race     mean_p
   <chr> <chr>     <dbl>
 1 young asian     0.277
 2 young black     0.279
 3 young other     0.281
 4 young hispanic  0.282
 5 old   other     0.382
 6 old   asian     0.383
 7 old   hispanic  0.385
 8 old   black     0.386
 9 young white     0.499
10 old   white     0.616

Shake and Rake

Now race

# Not showing calculation, since they're identical.
sample_data %>% group_by(race) %>% summarize(mean_weight = mean(raked_weight))

# A tibble: 5 × 2
  race     mean_weight
  <chr>          <dbl>
1 asian          6.68 
2 black          1.87 
3 hispanic       4.22 
4 other          1.14 
5 white          0.731

# Notice: slightly further from universe than after first iter
sample_data %>% group_by(age) %>% summarize(mean_weight = mean(raked_weight))

# A tibble: 2 × 2
  age   mean_weight
  <chr>       <dbl>
1 old         0.741
2 young       1.50 

... and repeat until sample means on all marginals match population ones.

Less Manual: with autumn

autumn is a wonderful package for raking.

library(autumn)

target <- list(
  age = c(young = .5, old = .5),
  race = c(white = .6,
             hispanic = .1,
             black = .1,
             asian = .1,
             other = .1)
)
# See ?harvest for summary of other functionality
sample_data <- sample_data %>% harvest(target,max_weight = 10)

# With raked weights, we recover the population mean as expected
sample_data %>% summarize(mean_p = weighted.mean(enjoys_pineapple,weights))

# A tibble: 1 × 1
  mean_p
   <dbl>
1  0.467

Back to reality

Sadly, reality makes weighting much harder.

• Phone response rates are now frequently less than 1%
• Many survey researchers now use online or other non-probability sample methods
• Outcomes are frequently related to propensity to respond in complicated ways
• So, respondents are getting increasingly weird, in ways we can’t just weight out along a small set of census demographics

In other words, we live in a world of Non-Ignorable Nonresponse. What effects does this have on weighting?

A huge amount of pressure on weighting

Raking on everything isn’t free

• In many cases, raking on everything we want to won’t even converge.
• So we end up picking and choosing which variables and interactions theoretically matter most.

Second class of issues: variance problems

• Raking on more dimensions can lower bias, but it often adds variance too
• This leads to heuristic ways to control variance
• Rules of thumb like desiring a \(deff_{kish}\) < 1.5
• Trimming weights (cap/floor the tails of the weights distribution)

Restating problems with raking on hard mode

Let’s restate the last few slides in terms of specific problems alternative weighting methods can solve:

1. “Making better weights” feels incredibly heuristic with vanilla raking
   1. How do we select variables and interactions?
   2. How do we trade off bias for variance?
2. Variables are either “in” or “out”- there’s no way to say e.g. “race x education matters more to me than gender x race, but include both”
3. It’s not clear in what sense the raking solution is optimal

Regularized Raking

(Barratt, Angeris, and Boyd 2021) show that the broader representative sample weights problem can be understood as an optimization problem:

\[ \begin{array}{ll} \operatorname{minimize} & \ell\left(f, f^{\mathrm{target}}\right)+\lambda r(w) \\ \text { subject to } & \quad w \geq 0, \quad \mathbf{1}^T w=1 \end{array} \] Let’s step through this term by term.

Regularized Raking - Loss

\[ \begin{array}{ll} \operatorname{\color{grey}{minimize}} & \ell\left(f, f^{\mathrm{target}}\right)\color{grey}{+\lambda r(w)} \\ \color{grey}{ \text { subject to } } & \color{grey}{ \quad w \geq 0, \quad \mathbf{1}^T w=1} \end{array} \]

• We have \(f\) and \(f^{target}\), which are functions of our sample and target population.
• We’ll also specify some loss function \(\ell\), which clarifies how we want \(f\) and \(f^{target}\) to be close

Regularized Raking

Loss examples - Ok, so what?

Regularized Raking

Loss examples - Getting Interesting

Regularized Raking

Loss examples - Now We’re Getting Somewhere

Regularized Raking

Loss examples - full force

Regularized Raking

Regularization

\[ \begin{array}{ll} \operatorname{\color{grey}{minimize}} & \color{grey}{\ell\left(f, f^{\mathrm{target}}\right)+}\lambda r(w) \\ \color{grey}{ \text { subject to } } & \color{grey}{ \quad w \geq 0, \quad \mathbf{1}^T w=1} \end{array} \]

Regularized Raking

Regularization Example 1

\[ r(w) = \sum_{i=1}^{n} w_i \log w_i \]

• This is the negative entropy, equivalent to the Kullback–Leibler divergence from the weights distribution \(w\) and the uniform distribution.
• This would express a desire that our weights be as uniform as possible, subject to all our other constraints
• This is the other half of what vanilla raking does!

Regularized Raking

Regularization Example 2

\[ D_{\text{KL}}(w \parallel w^{des}) = \sum_{i=1}^{n} w_i \log\left(\frac{w_i}{w_i^{des}}\right) \]

• Alternatively, might want a weights distribution close to some target distribution \(w^{des}\), where closeness is defined in terms of KL divergence.
• Motivations: minimizing extreme tails, more demand for smoothness

Regularized Raking

Constraints

\[ \begin{array}{ll} \operatorname{\color{grey}{minimize}} & \color{grey}{\ell\left(f, f^{\mathrm{des}}\right)+\lambda r(w)} \\ \text { subject to } & \quad w \geq 0, \quad \mathbf{1}^T w=1 \end{array} \]

• Non-negative weights are useful (not all estimators play well with negative weights)
• We want weights to sum to 1 (equivalently: rescale to sum to n)

So what does vanilla raking do?

\[ \ell\left(f, f^{\mathrm{target}}\right)=\left\{\begin{array}{ll} 0 & f=f^{\mathrm{target}}, \\ +\infty & \text { otherwise }, \end{array} \quad r(w)=\sum_{i=1}^n w_i \log w_i\right. \]

• Equality loss on all raking dimensions
• Negative entropy regularizer
• Hopefully, this gives some clarity about what raking optimizes for
• As we’ve seen, this is far from the only choice of \(\ell\) and \(r(w)\); are other choices better?

Data

Introducing regrake

Let’s walk through a basic example in both survey (for vanilla raking) and regrake (our R implementation of regularized raking based on (Barratt, Angeris, and Boyd 2021)). We’ll weight just on age, gender, race, and region to show syntax.

Vanilla raking with survey

formula_1 <- ~recode_age_bucket + recode_female + recode_race + recode_region

# Convenience function to make targets for a formula given a `survey` design obj
target_1 <- create_targets(cces_wt_design, formula_1)

pew_raked_1 <- calibrate(design = pew_design,
                       formula = formula_1,
                       population = target_1,
                       calfun = "raking")

svycontrast(svymean(~recode_vote_2016, pew_raked_1, na.rm = TRUE),
                        vote_contrast)

           nlcon     SE
contrast 0.08901 0.0259

Now with regrake

# With regrake, we directly specify what variables to match
# and use the CCES survey design as our population target
regrake_result_1 <- regrake(
  data = pew,
  formula = ~ rr_exact(recode_age_bucket) + rr_exact(recode_female) +
              rr_exact(recode_race) + rr_exact(recode_region),
  population_data = cces_wt_design,
  pop_type = "survey_design",
  regularizer = "entropy",
  lambda = 1
)

Steps:

1. Specify the formula with constraint types (rr_exact() for exact matching)
2. Choose a regularizer (entropy = prefer uniform weights)
3. Set the regularization strength (\(\lambda\))

Illustration: this is what raking is

n_pew <- nrow(pew)
regrake_weights_1 <- regrake_result_1$weights  # Already sums to n

regrake_design_1 <- svydesign(ids = ~1, data = pew, weights = regrake_weights_1,
                calibrate.formula = formula_1)

svycontrast(svymean(~recode_vote_2016, regrake_design_1, na.rm = TRUE),
                        vote_contrast)

            nlcon     SE
contrast 0.089006 0.0257

Same result as vanilla raking! With entropy regularizer and \(\lambda = 1\), we’re essentially doing the same optimization.

Illustration: weight distributions

Nearly identical weight distributions - as expected when we’re solving the same problem.

About as far as raking will take us

formula_2 <- ~ recode_age_bucket + recode_female +
    recode_region * recode_educ_3way * recode_race

# Convenience function to make targets for a formula given a `survey` design obj
target_2 <- create_targets(cces_wt_design,formula_2)

pew_raked_2 <- calibrate(design = pew_design,
                       formula = formula_2,
                       population = target_2,
                       calfun = "raking")

svycontrast(svymean(~recode_vote_2016, pew_raked_2, na.rm = TRUE),
                        vote_contrast)

            nlcon     SE
contrast 0.011029 0.0271

Mixed loss types

With regularized raking, we can do something vanilla raking can’t: exact matching on main effects, but soft (L2) matching on interactions.

This lets us include more information without the convergence problems that come from trying to exactly match every cell.

Mixed loss types: the code

# Main effects: exact matching (rr_exact)
# Interactions: L2/soft matching (rr_l2)
regrake_result_2 <- regrake(
  data = pew,
  formula = ~ rr_exact(recode_age_bucket) + rr_exact(recode_female) +
              rr_exact(recode_inputstate) + rr_exact(recode_region) +
              rr_exact(recode_educ) + rr_exact(recode_race) +
              rr_l2(recode_region:recode_educ) +
              rr_l2(recode_region:recode_race) +
              rr_l2(recode_educ:recode_race) +
              rr_l2(recode_region:recode_educ:recode_race),
  population_data = cces_wt_design,
  pop_type = "survey_design",
  regularizer = "entropy",
  lambda = 10
)

51 states + 6 main effect vars + all 2/3-way interactions = lots of constraints!

Results: weight distributions

regrake produces smoother weights - less extreme values.

Results: vote margin estimate

regrake_design_2 <- svydesign(ids = ~1, data = pew, weights = regrake_weights_2,
                calibrate.formula = ~recode_age_bucket + recode_female +
     recode_region + recode_educ + recode_race)

svycontrast(svymean(~recode_vote_2016, regrake_design_2, na.rm = TRUE),
                        vote_contrast)

            nlcon     SE
contrast 0.020404 0.0282

Checking the balance: what did we actually achieve?

Unlike vanilla raking, soft constraints don’t guarantee exact balance. The balance data frame lets us inspect what regrake actually achieved:

Exact constraints (blue) cluster at 0; L2 constraints (red) have slack.

Versus other modern options

MRP (and friends!)

Multilevel Regression and Postratification (Gelman et al. 2019) uses a regularized model to predict the outcome, and poststratify.

Versus other modern options

Multilevel Calibration

Multilevel Calibration (Ben-Michael, Feller, and Hartman 2023) also takes an optimization approach, requires exact matching on the marginals, more flexibility on interactions

I’ve only been talking about weighting today, so want to emphasize:

• Good sampling matters (and more than how you weight)
• Selection of weighting variables matters (and more than how you weight)

That said, how you weight is entangled with these, and more flexible weighting gives you more options with the above.